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(54) System for determining the spatial position and/or orientation of one or more objects 



(57) A system (1 0, 1 0') for determining spatial posi- 
tion and/or orientation of one or more objects (18, 18', 
22, 22') includes an optical subsystem (14, 14') and a 
non-optical subsystem, (12, 12'). The optical subsystem 
includes optical subsystem light sources (16a, 16b, 16c; 
16a', 16b', 16c0 mounted to one or more of the objects 
and an optical subsystem sensor (20, 20') adapted to 
detect energy from the optical subsystem light sources. 
The optical subsystem has an optical subsystem coordi- 
nate system (X Q , Y 0 , Zq) in a fixed relationship with the 
optical subsystem sensor. The optical subsystem sen- 
sor produces position and/or orientation signals relative 
to the optical subsystem coordinate system in response 
to optical subsystem light source detected energy. The 
non-optical subsystem has a non-optical coordinate 
system (X M , Y M , Z M ; X A , Y A , Z A ) and is adapted to pro- 
duce position and/or orientation signals of one or more 
of the objects (22, 22') relative to the non-optical sub- 
system coordinate system. A coupling arrangement (24, 
24'; 34, 34') is provided for producing position and/or 
orientation signals indicative of the position and/or ori- 
entation of the selected one of the optical or non-optical 
subsystems relative to the coordinate system of the 
other one of the optical or n on -optical subsystems. A 
processor (26, 26') is responsive to signals produced by 
the coupling arrangement and the individual subsystem 
sensors for determining the position and/or orientation 
of one or more of the objects relative to some conven- 
iently defined coordinate system. 
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Description 
TECHNICAL FIELD 

[0001] This invention relates generally to systems 
capable of determining the spatial position and angular 
orientation (i.e. pose) of three-dimensional (3D) bodies 
or objects, and more particularly to systems capable of 
tracking objects in real time within a specified volume, 
without regard to the objects' rigidity or visibility. 

BACKGROUND 

[0002] As is known in the art, a wide variety of sys- 
tems have been developed that can determine the spa- 
tial position and angular orientation of objects over 
small time-scales, and thus track their motion in real 
time. These systems generally make use of specific 
physical phenomena, and as such, have different capa- 
bilities and limitations. One system is an optical system. 
Such an optical system operates on sensing of sources 
of radiated electromagnetic energy (e.g., light or infra- 
red energy emitted from active markers or light or infra- 
red energy reflected from passive markers) by sensitive 
arrays, such as charge-coupled devices (CCD). Such 
optical systems can provide highly accurate spatial and 
angular measurements at high sampling frequencies 
over large operational volumes (typically room size), but 
require that a minimum number of the markers always 
be in view of the CCD sensors. This line-of-sight limita- 
tion can be partially remedied by determining the posi- 
tion of an obscured point from the measured positions 
of the visible markers with triangulation techniques. For 
example, markers can be affixed to instruments or 
probes (such as surgical probes) such that their tip 
points can be tracked. However, such probes must be 
rigid. This method cannot be applied to flexible probes 
such as catheters. 

[0003] Another type of system is a non-optical sys- 
tem. Such systems include magnetic systems, mechan- 
ical systems, and ultra-sonic systems. For example, 
U.S. Pat. No. 5,197,476 and 5,295,483 to Nowacki and 
Horbal disclose the use of optical cameras to track the 
position of an ultrasonic scanner or probe, which itself 
detects concretions such as kidney stones within the 
human body. The ultrasonic scanner cannot determine 
the tracked object's pose, though. Magnetic systems do 
not suffer the line-of-sight problem inherent in optical 
systems; but such systems can be severely affected by 
extraneous objects perturbing their magnetic fields, and 
are also generally less accurate. Mechanical systems 
use mechanical devices, such as articulating arms, and 
are free of line-of-sight and magnetic disturbance prob- 
lems; but such systems are considerably more expen- 
sive for a given level of accuracy. Their accuracies are 
subject to perturbations arising from gravitationally 
induced forces and torques, which limit them to the gen- 
erally smaller operational volumes spanned by their 



range of motion. Also, they are more cumbersome than 
other devices since their motion is constrained by possi- 
ble collisions with other objects lying within their opera- 
tional volume. 

5 [0004] As is also known in the art, optical devices 
can be used in conjunction with non-optical devices to 
overcome the optical devices' line-of-sight-limitations, 
but such coupling inconveniently results in the meas- 
ured position data being reported in separate frames of 

w reference, thus requiring the data to be reconciled by 
the user calibrating the devices to determine the neces- 
sary transformation between the two frames of refer- 
ence. Birkfellner, et. al., Concepts and Results in the 
Development of a Hybrid Tracking System for CAS, LeCm 

' 5 ture Notes in Computer Science: Medical image Computing and Com- 
puter-Assisted Intervention-MICCAI '98* Vol. 1496 (1998), pp. 
342-351 , describe such a system comprised of an opti- 
cal tracking system and a direct current pulsed electro- 
magnetic tracking system. They also describe 

20 procedures for calibrating and registering the magnetic 
system local frame of reference to the optical system 
frame of reference. Their system requires that the mag- 
netic field source remain fixed after the lengthy calibra- 
tion and registration procedures have been done, thus 

25 discouraging the movement of the field source to other 
convenient or appropriate positions as may be desired. 
Also, because their system reports position data from 
the magnetic subsystem only when the optical position 
data is unavailable because of obstructions in the opti- 

30 cal system's line-of-sight, it remains essentially an opti- 
cal system that is augmented by a non-optical system. 
[0005] U.S. Patent 5,831,260 to Hansen teaches a 
hybrid motion tracker having magnetic and optical sub- 
systems. This system is used for motion capture, using 

35 sensor assemblies (having both magnetic field sensors 
and optical Light Emitting diode (LED) sources) placed 
strategically on the person(s) being tracked to detect the 
motion. In normal operation, the optical sub-system pro- 
vides the 3D position data, because of its inherently 

40 greater accuracy, and the magnetic sub-system pro- 
vides the orientation data; if the optical sources are 
obscured, then the magnetic sub-system also provides 
the position data. Hansen's system is similar to the sys- 
tem described by Birkfellner, et. al, being primarily a 

45 coupling of a commercially available magnetic sub-sys- 
tem with a commercially available optical system, and 
using commercially available software to transform 
measurements between the subsystems (although 
some integration exists, such as using the optical sub- 
so system to compensate the magnetic sub-system's 
degrading Signal-to-Noise ratio). Thus Hansen's sys- 
tem suffers the same deficiencies inherent in such sys- 
tems, requiring fixed magnetic transmitters and fixed 
optical sensors whose fixed frames of reference must 

55 be first registered by the user by means of lengthy cali- 
bration and registration procedures, thereby precluding 
any easy repositioning of the sub-systems relative to 
one another as conveniently desired. 
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SUMMARY 

[0006] In accordance with the present invention, a 
system is provided for determining spatial position 
and/or orientation of one or more objects. The system 5 
includes an optical subsystem and a non-optical sub- 
system. The optical subsystem includes optical subsys- 
tem light sources mounted to one or more of the objects 
and an optical subsystem sensor adapted to detect 
energy from the optical subsystem light sources. The 10 
optical subsystem has an optical subsystem coordinate 
system in a fixed relationship with the optical subsystem 
sensor. The optical subsystem sensor produces posi- 
tion and/or orientation signals relative to the optical sub- 
system coordinate system in response to optical 15 
subsystem light source detected energy. The non -opti- 
cal subsystem has a non -optical coordinate system and 
is adapted to produce position and/or orientation sig- 
nals of one or more of the objects relative to the non- 
optical subsystem coordinate system. A coupling 20 
arrangement is provided for producing position and/or 
orientation signals indicative of the position and/or ori- 
entation of the selected one of the optical or non-optical 
subsystems relative to the coordinate system of the 
other one of the optical or non-optical subsystems. A 25 
processor is responsive to signals produced by the cou- 
pling arrangement and the optical subsystem and/or 
non-optical subsystem for determining the position 
and/or orientation of one or more of the objects relative 
to the coordinate system of the selected one of the opti- 30 
cal or non-optical subsystems. 

[0007] In one embodiment of the invention, the cou- 
pling arrangement comprises a non-optical subsystem 
light source having a fixed relationship to the non-optical 
subsystem coordinate system. The optical subsystem 35 
sensor is adapted to detect energy from the non-optical 
subsystem light source and produce position and/or ori- 
entation signals in response to such non-optical subsys- 
tem light source detected energy relative to the optical 
subsystem coordinate system. The processor is 40 
responsive to signals produced by the optical and/or 
non-optical subsystems for determining the position 
and/or orientation of one or more of the objects relative 
to the optical subsystem coordinate system. 
[0008] In another embodiment of the invention, the 45 
coupling arrangement comprises a non-optical subsys- 
tem sensor having a fixed relationship to the optical sub- 
system coordinate system. The non-optical subsystem 
sensor is adapted to produce position and/or orientation 
signals of the optical subsystem light source relative to so 
the non -optical subsystem coordinate system. The 
processor is responsive to signals produced by the opti- 
cal subsystem sensor and/or the non-optical system 
sensor for determining the position and/or orientation of 
one or more of the objects relative to the coordinate sys- 55 
tern of the non-optical system. 

[0009] With such an arrangement, a hybrid opti- 
cal/non-optical tracking system is provided which oper- 



ates in conjunction with a processor, that can measure 
and track a given 3D object's position and pose through- 
out the system's operational volume, both when the 
object is visible and when it is obscured. Furthermore, 
the object being tracked is not required to be a rigid 
body; nonrigid bodies such as catheters can be used. 
The ability of the present invention to track obscured 
nonrigid bodies is an important advantage over current 
optical systems. Further advantages imparted to non- 
optical devices that arise from their being coupled to 
optical devices depend on the specifics of the non-opti- 
cal device. 

[001 0] The hybrid system also has the advantage of 
not requiring calibration and coordinate system align- 
ment at the time of application. In addition, the hybrid 
system would still retain flexibility of arrangement - it 
would not be necessary to lock the subsystem compo- 
nents rigidly in place to maintain a fixed relative separa- 
tion. For example, a non-optical subsystem could be 
repositioned within the optical subsystem operational 
volume as desired to take advantage of optimum loca- 
tions or to avoid disturbances affecting its performance. 
[0011] The measurement redundancy inherent in 
hybrid systems provides them with a further advantage 
over their subsystem constituents. Position tracking sys- 
tems generally have accuracies that vary considerably 
throughout their operational volumes, being dependent 
on a wide variety of factors. The spatial dependence of 
the accuracy differs considerably between optical and 
non-optical devices, and typically one will be substan- 
tially more accurate than the other for a given region of 
space. Using measured accuracy mappings as a guide, 
hybrid systems can be designed that select the more 
accurate of the two valid measurements for a given 
operational subspace, or apply an appropriate weighted 
averaging, thus extending the overall region encom- 
passing a desired accuracy level, compared to the 
equivalent regions obtained from the optical and non- 
optical subsystems alone. In one embodiment of the 
invention, the non-optical subsystem is a magnetic 
tracking system and in another embodiment of the 
invention the non-optical subsystem is a mechanical 
tracking system. 

[0012] The details of one or more embodiments of 
the invention are set forth in the accompanying draw- 
ings and the description below. Other features, objects, 
and advantages of the invention will be apparent from 
the description and drawings, and from the claims. 

DESCRIPTION OF DRAWINGS 

[0013] 

FIG. 1 is a schematic diagram of a hybrid, opti- 
cal/non-optical, magnetic, system adapted to deter- 
mine the spatial position and angular orientation 
(i.e. pose) of three-dimensional (3D) bodies or 
objects according to the invention. 
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FIG. 1A is a schematic diagram of an alternative 
embodiment of a hybrid, optical/non -optical, mag- 
netic, system adapted to determine the spatial posi- 
tion and angular orientation (i.e. pose) of three- 
dimensional (3D) bodies or objects according to the 
invention. 

FIG. 1 B is a schematic diagram of another alterna- 
tive embodiment of a hybrid, optical/non-optical, 
magnetic, system adapted to determine the spatial 
position and angular orientation (i.e. pose) of three- 
dimensional (3D) bodies or objects according to the 
invention. 

FIG. 2 is a schematic diagram of a hybrid, opti- 
cal/non-optical system adapted to determine the 
spatial position and angular orientation (i.e. pose) 
of three-dimensional (3D) bodies or objects accord- 
ing to another embodiment of the invention. 
FIG. 3 is a process flow diagram describing the cal- 
ibration of the hybrid, optical/non -optical system of 
FIG. 1A. 

FIGS. 4A, 4B and 4C together depict a process flow 
diagram describing the basic operation of the 
hybrid, optical/magnetic system of FIG 1 A. 

[0014] Like reference symbols in the various draw- 
ings indicate like elements. 

DETAILED DESCRIPTION 

[0015] Referring now to FIGS. 1 and 2, we present 
two embodiments of a system for determining spatial 
position and/or orientation of one or more objects. In 
each embodiment, the system 10, 10*, respectively, 
includes a n on -optical subsystem 12, 12', respectively, 
and an optical subsystem 14, 14', respectively. In sys- 
tem 1 0 (FIG. 1 ), the non-optical subsystem 1 2 is a mag- 
netic tracking system and in system 10' (FIG. 2), the 
non-optical subsystem 12' is a mechanical tracking sys- 
tem. In both systems 10 and 10', the optical subsystem 
14, 14', respectively, includes: an optical subsystem 
light source, here three light sources 16a, 16b, 16c, 
mounted to one of the objects, here to object 18; and, a 
sensor 20 adapted to detected energy from the optical 
subsystem light sources 16a, 16b, 16c. The optical sub- 
system 14 has an optical subsystem coordinate system 
Xq, Y q , Zq in a fixed relationship with the sensor 20. 
The sensor 20 produces position and/or orientation sig- 
nals in response to detected energy from the optical 
subsystem light sources 16a, 16b, 16c relative to the 
optical subsystem coordinate system Xq, Y q , Zq. 
[0016] The non-optical subsystems 12, 12' each 
have a non-optical subsystem coordinate system X^, 
Y M , Z M and X A , Y A , Z A , respectively as indicated, and 
are each adapted to produce position and/or orientation 
signals of another one of the objects 22, 22', respec- 
tively, relative to the no n -optical subsystem coordinate 
system X M , Y M , Z M and X A , Y A , Z A , respectively. The 
non-optical subsystems 12, 12' also include a non-opti- 



cal subsystem light source, here a plurality of coupling 
markers 24, 24', as indicated, having a fixed relationship 
with the non-optical subsystem coordinate system X M , 
Y M , Z M and X A , Y A , Z A , respectively. The sensor 20, 20' 

5 of the optical subsystem 14, 14' is adapted to detect 
energy from the non-optical subsystem light source 24, 
24', respectively, and produce position and/or orienta- 
tion signals in response to the non-optical subsystem 
detected energy relative to the optical subsystem coor- 

70 dinate system Xq, Y q , Zq- Thus, the spatial position of 
the non-optical coordinate system X M , Y M , Z M and X A , 
Y Al Z A relative to the optical subsystem coordinates Xq, 
Y Q , Zq are determinable by the detection, and process- 
ing, of the light energy produced by the non-optical sub- 

15 system light sources 24 by sensor 20, which sensor is in 
fixed relationship with the optical subsystem coordinate 
system Xq, Yq, Zq. A processor 26, 26' here a host 
computer, is responsive to signals produced by the sen- 
sor 20, 20* and determines the position and/or orienta- 

20 tion of the objects 18, 22 and 18', 22' relative to the 
optical subsystem coordinate system X Q , Y G , Zq. 
[0017] Thus, from the above, in FIGS. 1 and 2, the 
light sources 24, 24', respectively, and the sensors 34, 
34' provide a coupling arrangement for producing posi- 
es tion and/or orientation signals indicative of the position 
and/or orientation of the selected one of the optical or 
n on -optical subsystems, here the non -optical coordi- 
nate systems X M , Y M , Z M and X A , Y A , Z A , respectively, 
relative to the coordinate system of the selected one of 

30 the other one of the optical or non -optical subsystems, 
here the optical coordinate systems Xq, Y q , Zq. 
In both embodiments, the hybrid system 10, 10* 
includes a single non-optical subsystem 12, 12' coupled 
to an optical subsystem 14. This is for illustrative pur- 

35 poses only, and it is understood that such hybrid sys- 
tems are more general, so that hybrid systems having 
more than one type of non-optical subsystem could be 
equally well coupled to the optical subsystem in an 
appropriate manner. 

40 [0018] Referring now in more detail to FIG. 1, the 
hybrid, optical/non -optical, tracking system 1 0 is shown. 
Here, the non-optical subsystem 1 2 is, as noted above, 
a magnetic position tracking system. The non-optical 
subsystem 12 is coupled to an optical subsystem 14 of 

45 the tracking system through the processor 26 and, as 
noted above, through the optical link between the non- 
opticai subsystem light sources 24 and the sensor 20. 
Magnetic tracking systems are well known, and several 
variants have been developed. For illustrative purposes, 

so here the non-optical, here magnetic, subsystem 12 of 
the tracking system 10 includes a magnetic field gener- 
ator 28 comprised of suitably arranged electromagnetic 
inductive coils, not shown, that serve as the spatial 
magnetic reference frame (i.e., is fixed relative to the 

55 non-optical subsystem coordinate system X M , Y M , Z M ). 
The non-optical subsystem 12 includes small mobile 
inductive sensor coils 30, which are attached to the 
object 22 being tracked. It should be understood that 
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other variants could be easily accommodated. The non- 
optical subsystem 12 also includes a magnetic system 
control unit 32 that is coupled to the processor 26, mag- 
netic field generator 28 and coils 30, as indicated 
[0019] More particularly, the magnetic field genera- 
tor 28 defines the magnetic coordinate reference frame 
(X M , Y M , Z M ). The generator 28 includes sufficient 
number of coils suitably arranged to generate the 
source magnetic field. A small sensor coil 30 is attached 
to the object 22, here a flexible element (such a device 
could be inserted into a catheter). The position and 
angular orientation of the coil 30 (i.e., pose) are deter- 
mined from its magnetic coupling to the source field pro- 
duced by magnetic field generator 28. The magnetic 
system control unit 32 manages the magnetic field gen- 
erator 28 and receives signals from the magnetic sen- 
sors, here the coil 30. 

[0020] It is noted that the magnetic field generator 
28 generates a sequence, or set, of here 6, different 
spatial magnetic field shapes, or distributions, each of 
which is sensed by the sensor coil 30. Each sequence 
enables a sequence of signals to be produced by the 
coil 30. Processing of the sequence of signals enables 
determination of the pose of the coil 30, and hence the 
pose of the object 22 to which the coil 30 is mounted rel- 
ative the magnetic (non-optical) coordinate reference 
frame X M , Y M , Z M that is in fixed relationship to the mag- 
netic field generator 28. As noted above, a plurality of 
non-optical subsystem light sources 24 (i.e., emitters) is 
mounted to, i.e., fixed to, the magnetic field generator 
28, and hence, in fixed relationship to the magnetic 
(n on -optical) coordinate reference frame X M , Y M , Z M . 
[0021] Similarly, optical tracking systems, such as 
optical subsystems 14 are well known, and several var- 
iants have been developed. Again, for illustrative pur- 
poses, here the optical subsystem 14 is an infrared 
system that includes the optical sensor 20, here two 
two-dimensional charge coupled devices (CCDs). The 
optical subsystem light sources 16a, 16b, 16c, (e.g., 
optical markers which may be passive reflectors of a 
source or active light emitting diodes (LEDs)) are 
mounted to object 1 8. It should be understood that other 
variants would be equally well suited. 
[0022] More particularly, the optical subsystem 14 
includes the optical position sensor 20, here comprised 
of two, two-dimensional CCDs 34. The optical subsys- 
tem light sources 16a, 16b and 16c as well as the non- 
optical subsystem light sources 24 are reflective type 
light sources (i.e., passive markers) and here directional 
infrared energy source LEDs 36 are used to illuminate 
the passive markers 1 6a, 1 6b, 1 6, and 24. The infrared 
energy sources 36 are arranged about the CCDs 34 in 
an annular fashion, as indicated. The optical position 
sensor 20 defines (i.e., is fixed relative to) the optical 
coordinate reference frame, Xq, Y 0 , Zq. The optical 
position sensor 20 is coupled to an optical subsystem 
control unit 40 that manages the optical data collection 
and timing, the firing of the directional energy sources 



36. It is noted that if active markers had been used for 
the optical subsystem light sources 16a, 16b, 16c and 
24, the optical subsystem control unit 40 would manage 
the optical data collection and timing, the firing of the 

5 active markers 16a, 16b, 16c and 24. The unit 40 also 
provides as other tasks necessary for optical tracking. 
[0023] The non -optical and optical (magnetic) sub- 
systems 12, 14 are linked together through their con- 
nection to the processor 26, which synchronizes the 

10 operations of the two subsystems and manages the 
overall data acquisition. To provide a common reference 
frame for the overall system 10, the magnetic field gen- 
erator 28 has a minimum of three active LED optical 
markers 24 shown attached to it in an arrangement well 

15 suited to define it as a rigid body to the optical subsys- 
tem 12 (it being understood that better accuracy and 
greater flexibility would be achieved by increasing the 
number of reference markers, i.e., light sources). The 
non-optical subsystem light sources 24 coordinates in 

20 the non-optical (magnetic) subsystem coordinate sys- 
tem (i.e., reference frame) X M , Y M , Z M would be accu- 
rately determined at the time of manufacture, when the 
system 1 0 is calibrated. Various methods can be used 
for the determination, such as the numerical fitting of a 

25 suitably large set of three-dimensional (3D) measure- 
ments obtained individually by the non-optical (mag- 
netic) and optical subsystems 12, 14 within their own 
frames of reference, relative to some third calibration 
standard, or through the use of hybrid tools comprised 

30 of optical markers and magnetic sensors combined in a 
suitable arrangement. 

[0024] With the positions of the non-optical light 
sources 24 (i.e., the magnetic field reference markers) 
known relative to the non-optical coordinate system X M , 

35 Y M , Z M , the positions and poses of the object 22 being 
tracked by the overall hybrid system 10 can be reported 
in a common reference frame, here the common refer-, 
ence being the optical subsystem coordinate system 
Xo Y 0 , Zq, fixed relative to the optical position sensor 

40 20. 

It should be understood that the non-optical coordinate 
reference X M , Y M , Z M can also be coupled to the optical 
reference frame X 0 , Y Q , Zq by having a magnetic sen- 
sor 30a (Fig. 1 B) fixed to the optical position sensor 20 
45 relative to the non-optical coordinate reference X M , Y M , 

If the magnetic subsystem reference frame X M , Y M , Z M 
is chosen as the common reference frame, then the 
transformation TOM will convert measurements in the 

so optical subsystem coordinate system X Q , Yq, Zq to their 
equivalent non-optical subsystem coordinate system 
X M , Y M , Z M . Such transformation methods are well 
known, and are completely general; the common refer- 
ence frame need not be limited to either of the subsys- 

55 tern (reference frames) Xq, Y 0 , Zq or X M , Y M , Z M , but 
can be any convenient arbitrary frame. The transforma- 
tions are done on the processor 26 in this implementa- 
tion, but this is not required. The magnetic and optical 
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system control units 32, 40 could also be linked together 
to exchange data directly prior to position determina- 
tions and transformations. Either way, the combined 
data from the two subsystems 12, 14 provides advan- 
tages otherwise not available. For example, the optically 
measured position can be used to initialize the magnetic 
position determination algorithm, decreasing the calcu- 
lation time considerably. 

[0025] The optical tracking of the magnetic field 
generator 28 with non-optical light sources 24 in this 
embodiment has the advantage of allowing the mag- 
netic field generator 28 to be optimally repositioned with 
respect to the tracking volume to avoid interference from 
magnetic and conductive objects, or to obtain the best 
performance of the magnetic subsystem. Note that the 
tracking procedure need not be interrupted while the 
position of the magnetic field generator 28 is changed. 
[0026] Referring now to FIG. 1A, a hybrid optical- 
magnetic probe 41 is shown in the system 10 of FIG. 1, 
(where like elements are designated by the same 
numerical designation as used in FIG. 1). Such probe 
41 has a magnetic sensor, here a coil 30b, located at its 
end position, as well as three visible active optical sub- 
system light sources 43 detectable by sensor 20 (FIG. 
1 ) that can be used to define the probe as a rigid body, 
and thus simultaneously track the end position with the 
magnetic sensor 30b. Such a probe 41 can be used as 
a reference, for example, to detect possible magnetic 
disturbances that could affect the magnetic measure- 
ments. 

[0027] Although this embodiment shows only a sin- 
gle magnetic field generator 28, greater flexibility and 
extension could be achieved by using multiple field gen- 
erators, each separately tracked with its own set of non - 
optical subsystem light sources 24 (FIG. 1), i.e., optical 
markers. The easiest method would be to optically track 
individual field generators 28, which could be conven- 
iently located as appropriate throughout the optical sub- 
system 12 operational volume, thus greatly enhancing 
the magnetic subsystem operational volume. This 
would be equivalent to a customizable magnetic field 
generator that could be optimally arranged to best 
accommodate the constraints inherent to a specific 
measurement situation. For example, in a surgical appli- 
cation, the coils could be appropriately arranged over 
the patient's body to greatly extend the operational 
range of a catheter containing the magnetic sensor. 
[0028] As noted above, the coupling between the 
optical and magnetic subsystems 14, 12 (FIG. 1) in this 
embodiment through the use of non -optical subsystem 
light sources 24 affixed to the magnetic field generator 
28 could be augmented by also affixing magnetic sen- 
sors such as coil 30a onto the optical sensor 20 itself, as 
shown in FIG. 1B (where like elements are designated 
by the same numerical designation as used in FIG. 1) 
thereby allowing the alignment of the individual non- 
optical (magnetic) and optical subsystem coordinate 
systems (X M , Y M , Z M and Xq, Y q , Zq, respectively, even 



when no direct line of sight exists between the two. This 
would require that the optical sensor 20 be housed in a 
property shielded casing made from materials that 
would not interfere with the magnetic fields, or that any 

5 influences from electrically conducting or magnetic 
materials be compensated by other means. 
[0029] Thus, from the above, in FIG. 1B, the coil 
30a has a fixed relationship to the optical subsystem 
coordinate system and is adapted to produce position 

w and/or orientation signals of the optical subsystem sen- 
sor 20 relative to the non-optical subsystem coordinate 
system. Thus, here the coil 30a provides a coupling 
arrangement for producing position and/or orientation 
signals indicative of the position and/or orientation of 

15 the selected one of the optical or non-optical subsys- 
tems, here the optical coordinate system Xq, Yq, Zq rel- 
ative to the coordinate system of the other one of the 
optical or non-optical subsystems, here the non-optical 
coordinate systems X M , Y M , Z M . 

20 [0030] Referring now to FIG. 2, the hybrid opti- 
cal/non-optical tracking system 10' is shown. It is noted 
that like elements used herein and used in the system 
10 of FIG. 1 are designated with the same numerical 
designation. As noted above, the non -optical subsystem 

25 12' is a mechanical tracking subsystem, here an articu- 
lating arm 50'. The arm 50' is coupled to the optical sub- 
system 14' of the tracking system 10' though the non- 
optical subsystem light sources 24' affixed to the arm 
50' and detection of light from the sources 24' by the 

30 optical position sensor 20'. Mechanical articulating arm 
systems are well known, and several variants have 
been developed. Smaller arms can have better accu- 
racy than optical systems, but are limited to operational 
volumes much smaller than typical optical system oper- 

35 ational volumes. Here, again, the optical subsystem 14' 
operates in the infrared region, this time consisting of an 
optical sensor 20' comprised of three one-dimensional 
CCDs 34; other variants would be equally well suited. 
Unlike the magnetic/optical embodiment already con- 

40 sidered in connection with FIG. 1 , this embodiment 
does not handle non-rigid bodies because of the funda- 
mental limitations inherent to articulating arms, but it 
does provide an example of the enhancement brought 
about to a non-optical device by coupling it to an appro- 

45 priate optical device. 

[0031] Thus, as shown in FIG. 2, the articulating 
arm 50' defines the non-optical (arm) subsystem coordi- 
nate system (reference frame) X A , Y A , Z A . The arm 50' 
includes, as noted above, a plurality of mechanically 

so linked rigid segments, stiffly connected, that can be 
rotated and moved about such that the end-effector 22' 
can be positioned as desired within the arm's opera- 
tional volume of motion. The positions and orientations 
of each segment are accurately determined and com- 

55 bined to give the end-effector 22' position relative to the 
arm's local reference frame (i.e., the non-optical sub- 
system coordinate system X A , Y A , Z A ). The arm's con- 
trol unit 32' manages the data acquisition from the arm 
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and interfaces the unit to the processor 26*. 
[0032] The optical subsystem 14* includes a posi- 
tion sensor 20' comprised of three one-dimensional 
CCDs 34\ The position sensor 20* defines the optical 
coordinate reference frame (Xq, Yq, Zq). An example 
rigid probe 18* with three visible active LED markers 
16a', 16b' 16c' is coupled to the control unit 40'. The 
optical position sensor 20' is attached to control unit 40* 
that manages the optical data collection and timing, the 
firing of the active marker LEDs, as well as other tasks 
necessary for optical tracking. 

[0033] The non-optical (mechanical) and optical 
subsystems 12' 14' are linked together through their 
connection to the processor 26', which synchronizes the 
operations of the two subsystems and manages the 
overall data acquisition, and through the detection of 
light from the non-optical subsystem light sources 24* by 
the optical position sensor 20', in order to provide a 
common reference frame for the overall system 10\ 
here the optical subsystem coordinate system Xq, Y 0 , 
Zq fixed relative to the sensor 20'. More particularly, the 
articulating arm 50' has a plurality of active LED optical 
markers (i.e., the n on -optical subsystem light sources 
24') attached to it in an arrangement well suited to 
define it as a rigid body to the optical subsystem 1 4'. 
The processor 26' handles the transformations from the 
arm's frame of reference (i.e., the non-optical subsys- 
tem coordinate system X A , Y A , Z A ) to the optical sub- 
system's 14" frame of reference (i.e., the optical 
subsystem coordinate system Xq, Yq, Zq). Thus the 
arm 50' can be relocated anywhere within the optical 
subsystem's 14' operational volume, thereby greatly 
extending the arm's effective operational volume. The 
position of the non-optical subsystem light sources 24* 
relative to the non-optical subsystem coordinate system 
X A , Y A , Z A would be accurately determined at the time 
of manufacture, when the system 1 0' is calibrated. 
[0034] With the positions of the articulating arm ref- 
erence markers (i.e., non-optical subsystem light 
sources 24') known, the position of the arm end-effector 
22' can be reported in a common reference frame, here 
the optical subsystem coordinate system Xq, Y q , Zq. 
Transformation methods for converting the local arm 
coordinates to the optical subsystem's frame of refer- 
ence (i.e., the optical subsystem coordinate system X Q , 
Yo Zq) are well known, and are completely general; 
again, the common reference frame need not be limited 
to either of the subsystem reference frames, but can be 
any convenient arbitrary frame. The transformations are 
done on the processor 26' in this implementation, but 
this is not required if the two subsystem control units 
32', 40' are connected so that they can directly 
exchange data. 

[0035] An example factory calibration of a mag- 
netic/optical hybrid system is illustrated in the flow dia- 
gram, FIG. 3. There is much flexibility in this process, in 
part because of the variety of other position measure- 
ment devices available for use as standards, and thus 



many other variations of the process described in FIG. 
3 would be well suited to the task. It is necessary that 
the standard device be substantially more accurate than 
either of the two subsystem devices over their respec- 

5 tive operational volumes; in this example, a coordinate 
measuring machine (CMM) is assumed to be the stand- 
ard. As stated in FIG. 3, Step 101, the optical subsys- 
tem is calibrated first. This is well known in the art and 
typically involves the accurate positioning of optical 

7 0 markers in some suitable grid-like fashion throughout 
the operational volume of the optical sensor 20 and 
measuring the sensor CCD data. The sensor data and 
CMM position data are then used to determine the sen- 
sor's operational parameters and local frame of refer- 

75 ence (Xq, Y q , Zq), which are subsequently used to 
convert the sensor data to their corresponding 3D posi- 
tion data during the sensor's normal operation. After the 
sensor parameters have been determined, the sensor 
data collected during the calibration can be converted to 

20 3D positions, and compared to the standard 3D posi- 
tions. The differences can be used to form a mapping of 
the sensor's spatial error distribution, which can be 
recorded for subsequent use in the hybrid system 3D 
position determination algorithms. 

25 [0036] As stated in FIG. 3, Step 102, the magnetic 
field generator 28 is then defined as a rigid body in the 
optical subsystem reference frame (Xq, Y q , Zq). Again, 
this is a well-known practice in the art, and typically 
involves measuring the 3D positions of the reference 

30 LED markers 24 affixed to the generator 28 relative to 
one another. The marker positions can be subsequently 
used to define a local coordinate system whose origin 
and orientation can be conveniently located within the 
rigid body. In this case, aligning the local rigid-body 

35 coordinate system with the local magnetic subsystem 
frame of reference (X M , Y M , Z M ) after it is determined is 
preferred. 

[0037] The magnetic subsystem is calibrated next, 
Step 1 03. The magnetic field generator 28 and the opti- 

40 cal sensor 20 are positioned such that each of their 
operational volumes lie within the CMM operational vol- 
ume and such that there is a clear line of sight between 
the field generator reference markers 24 and the optical 
sensor 20. The position and pose of the field generator 

45 28 is measured by the optical sensor 20 and recorded. 
A hybrid optical/magnetic sensing tool 41 having the 
magnetic sensor defined as a virtual marker, so that the 
optical and magnetic measurements are coincident, is 
affixed to the CMM end-effector and accurately posi- 

so tioned in a suitable grid-like manner throughout the 
magnetic subsystem operational volume, while the 
magnetic sensor data and optical sensor data corre- 
sponding to each magnetic sensor position are 
recorded. The magnetic sensor data and the CMM 3D 

55 position data are then used to determine the magnetic 
subsystem's operational parameters and local frame of 
reference, (X M , Y M , Z^), which are subsequently used 
to convert the magnetic sensor data to their correspond- 



7 



13 



EP 1 096 268 A2 



14 



ing 3D position data during normal operation. Similar to 
the optical subsystem calibration, after the magnetic 
subsystem parameters have been determined, the 
magnetic sensor data collected during the calibration 
can be converted to 3D positions and compared to the 
standard 3D positions so that their differences can be 
used to form a mapping of the magnetic subsystem's 
spatial error distribution. Note that although we have 
used a CMM to calibrate both subsystems, the optical 
subsystem itself could have been used instead to cali- 
brate the magnetic subsystem, provided it were suffi- 
ciently more accurate than the magnetic subsystem. 
[0038] The optical and magnetic 3D data corre- 
sponding to the hybrid tool positions, together with the 
field generator location and rigid-body data defined by 
the reference markers 24, can be used to calculate the 
required transformations between the optical subsys- 
tem and magnetic subsystem local frames of reference, 
(X 0 , Y 0 , Zq) and (X M , Y M , Z M ), respectively. The hybrid 
system global frame of reference can then be deter- 
mined in Step 104. The calculations used to determine 
these transformations are well known in the art. These 
transformations can be stored on either the system con- 
trol units 32 and 40, or the host computer 26, depending 
on the system design, and updated as required when- 
ever the field generator 28 or sensor device 20 is repo- 
sitioned. 

[0039] An example measurement process corre- 
sponding to the hybrid, optical/magnetic system 
described in FIG. 1A is illustrated in the flow diagram, 
FIGS. 4A, 4B, and 4C. In this example, a number of dif- 
ferent objects are tracked by the hybrid system. Some of 
these objects contain both optical markers and mag- 
netic sensors (such as the hybrid tool 41), while other 
objects contain only optical markers (such as the optical 
probe 18) or magnetic sensors (such as the magnetic 
sensor 30). As described in FIG. 4A, Step 201 , the mag- 
netic field generator 28 is conveniently positioned within 
the operational volume of the optical sensor 20 such 
that the reference marker LEDs 24 are visible to the 
optical sensor. 

[0040] The measurement process for each object 
being tracked begins in FIG. 4A, Steps 202, 203 with the 
optical sensor 20 determining the position of the mag- 
netic field generator 28 and thus the position of the 
magnetic subsystem local frame of reference (X M> Y M , 
Z M ) relative to the optical subsystem local frame of ref- 
erence (X 0 , Y 0 , Zq). If the field generator 28 has been 
moved relative to the optical sensor, the transforms 
between the subsystem tocal frames of reference and 
the global frame of reference are all updated to reflect 
the new positions and orientations (FIG. 4A, Steps 204 
and 205), otherwise, the determination of the object's 
position and pose begins (Step 206). 
[0041] The system first determines whether optical 
or magnetic data, or both types, are available. If only 
one type of data is present, either because the object 
being tracked has only one class of transducer (such as 



the optical probe 18 or the magnetic sensor 30), or 
because one type of data is invalid (such as the optical 
markers 43 on the hybrid tool 41 being obscured, or its 
magnetic sensor 30b being out of range), or simply 

5 because one type of data is unavailable (the optical and 
magnetic data collection frequencies could differ), then 
the measurement process branches to the appropriate 
subsystem (FIG. 4B, Step 207). For optical data, the 
optical subsystem determines the object's position and 

70 pose from the optical sensors in the local frame of refer- 
ence and then transforms the data to the global frame of 
reference (FIG. 4B, Steps 208 and 209), while for mag- 
netic data, the magnetic subsystem determines the 
object's position and pose from the magnetic sensors in 

is the subsystem's local frame of reference and then 
transforms the data to the global frame of reference 
(FIG. 4B, Steps 210 and 211). 

[0042] If both types of data are available, then the 
object's position and pose are determined by each sub- 

20 system in its local frame of reference and transformed to 
the global frame of reference in the usual manner, 
except that the subsystems can also exchange data 
between each other to assist the position and orienta- 
tion calculations by initializing their determination algo- 
us rithms, or to further refine calculated positions and 
orientations (FIG. 4C, Steps 212 to 216). Also, the 
uncertainties for each measurement are estimated from 
the subsystem error maps that were determined during 
factory calibration. The redundant measurements are 

30 then combined according to their estimated uncertain- 
ties to give a final position and pose, which is reported 
to the user (FIG. 4C, Step 217). Various methods of 
combining the redundant measurements are possible, 
including simply selecting the more accurate one, or 

35 weighted averaging according to uncertainty inverses. 
More sophisticated algorithms could also be used to 
recognize and discriminate anomalous measurements 
by thresholding differences between the subsystem 
measurements according to their respective uncertain- 

40 ties, so that, for example, a magnetic measurement per- 
turbed by the presence of a metal object near the 
sensors would be recognized by its significant differ- 
ence from the optical measurement and thus ignored. 
[0043] In view of the foregoing description of the 

45 two embodiments described above, it will now be recog- 
nized that, a hybrid optical/no n -optical system 10, 10' is 
provided for tracking the positions and orientations of 
three-dimensional (3D) objects, both rigid and non rigid, 
in space. The optical subsystem 14, 14' of the systems 

so 10, 10' have some appropriate form of light sensors 
(typically, three or more one-dimensional charge-cou- 
pled devices (CCDs), or two or more two-dimensional 
CCDs), a plurality of optical subsystem light sources 
(typically, active light emitting diode (LED) markers, or 

55 passive reflecting markers), and an electronic system 
control unit to manage the various operational aspects 
of the subsystem. The hybrid system includes an appro- 
priate non-optical subsystem, or subsystems that are 
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complementary to the optical subsystem in the sense 
that it is not subject to line-of -sight limitations. The non- 
optical subsystem includes, but is not limited to, mag- 
netic tracking devices, articulating mechanical arms, 
and optical fiber devices. The non-optical subsystem 5 
may include a plurality of at least three non-optical sub- 
system light sources properly affixed in known locations 
on some appropriate part of the device so as to form a 
rigid body that allows for coordinate transformations 
between the optical and non-optical subsystem coordi- 10 
nate reference frames. The hybrid system includes a 
processor interface and data management program that 
synchronizes the operation of the optical and non-opti- 
cal subsystems, performs the required subsystem 
measurement coordinate transformations to the hybrid- 15 
system's reference frame, and properly selects or deter- 
mines the best measurement in the case of redundant 
measurements from the two or more subsystems. 
[0044] Further, it is noted that the coupling between 
the optical and non-optical subsystems is generally 20 
such that the subsystems can be repositioned relative 
to one another to better take advantage of subsystem- 
specific optimum locations and to lessen the detrimental 
effects of subsystem-specific disturbances. Further, the 
reference optical markers affixed to the non-optical sub- 25 
system (s) are accurately determined at time of manu- 
facture, precluding the requirement of calibration and 
subsystem coordinate alignment being done at the time 
of application. The methods used for the said calibration 
depend on the specifics of the optical and non-optical 30 
subsystems. These include, but are not limited to, the 
numerical fitting of large sets of 3D position measure- 
ments obtained individually with the subsystems within 
their own frames of reference, relative to some calibra- 
tion standard, and the use of hybrid optical/non-optical 35 
calibration tools. 

[0045] Further, the individual subsystem control 
units responsible for determining object positions and 
orientations (within the subsystem local frame of refer- 
ence) through the use of appropriate calculation proce- 40 
dures are directly connected together to exchange the 
said position and orientation values in real-time so that 
transformations to a common hybrid system global 
frame of reference can be performed. In lieu of such a 
direct connection between the subsystems, the said 45 
transformations shall be calculated and applied on the 
processor. The optical and non-optical subsystems 
have their 3D positional and orientation al accuracies 
mapped as functions of their spatial locations at the time 
of manufacture to provide the basis for appropriate so 
selection or determination of the optimum measure- 
ment values in case of redundant measurements from 
the subsystems. 

[0046] With regard to the magnetic non-optical sub- 
system, it is noted that the magnetic subsystem 55 
includes one or more field generators and a number of 
magnetic sensors, one or more of which are affixed to 
the optical subsystem in such a manner as to allow the 



determination of the optical subsystem's frame of refer- 
ence within the magnetic subsystem's frame of refer- 
ence, thereby allowing the alignment of the two said 
frames of reference even when there is no line of sight 
between them. 

[0047] With regard to the magnetic non-optical sub- 
system, it is also noted that the magnetic subsystem 
can comprise a number of separate, individual electro- 
magnetic inductive field generator coils, each having at 
least three optical markers affixed to it in such a manner 
as to define it as a rigid body within the said optical sub- 
system's frame of reference, such that the individual 
coils can be placed anywhere within the said optical 
subsystem operational volume and tracked by the opti- 
cal sensors, thereby allowing the magnetic subsystem 
to be optimally adapted to track magnetic sensors over 
larger, more irregular sub-volume shapes and forms, 
than would be possible by a fixed array of generator 
coils localized within a single field generator unit. 
[0048] With further regard to the magnetic non-opti- 
cal subsystem, it should also be understood by one 
skilled in the art that the magnetic sub-system need not 
be restricted to an active field generator with passive 
magnetic sensors, but that its complement, a magnetic 
sub-system comprised of active magnetic "sensors" 
generating the fields, which are then detected by the 
passive "field generator* within its frame of reference, 
would also be suitable. 

[0049] A number of embodiments of the invention 
have been described. Nevertheless, it will be under- 
stood that various modifications may be made without 
departing from the spirit and scope of the invention. 
Accordingly, other embodiments are within the scope of 
the following claims. 

Claims 

1 . A system for determining spatial position and/or ori- 
entation of one or more objects, comprising: 

(A) an optical subsystem (14), comprising: 

(a) optical subsystem light sources (16a, 
1 6b, 1 6c) mounted to one or more of the 
objects (18); and, 

(b) an optical subsystem sensor (20) 
adapted to detect energy from the optical 
subsystem light sources (16a, 16b, 16c), 
such optical subsystem having an optical 
subsystem coordinate system (X Q , Y Q , Zq) 
in a fixed relationship with the optical sub- 
system sensor (20), such optical subsys- 
tem sensor (20) producing position and/or 
orientation signals relative to the optical 
subsystem coordinate system in response 
to optical subsystem light source detected 
energy; 
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(B) a non-optical subsystem (12) having a non- 
optical coordinate system (X Ml Y M , Z M ), such 
non-optical subsystem being adapted to pro- 
duce position and/or orientation signals of one 
or more of the objects (22) relative to the non- 
optical subsystem coordinate system; charac- 
terized by: 

(C) a coupling arrangement (24, 34) for produc- 
ing position and/or orientation signals indica- 
tive of the position and/or orientation of a 
selected one of the optical or non- optical sub- 
systems relative to the coordinate system of 
the other one of the optical or non-optical sub- 
systems; and 

(D) means (32, 40) responsive to signals pro- 
duced by the coupling arrangement (24, 34) 
and the optical and/or non-optical subsystem 
(12, 14) for determining the position and/or ori- 
entation of one or more of the objects (18, 22) 
relative to the coordinate system of the 
selected one of the optical or non-optical sub- 
systems (12, 14). 

2. A system according to claim 1 , characterized in that 
the optical light sources (43) are mounted to one of 
the objects (41) and the non-optical subsystem (12) 
is adapted to produce the position and/or orienta- 
tion signals of such one (41) of the objects. 

3. A system according to claim 1 , characterized in that 
the optical light sources (16a, 16b, 16c) are 
mounted to one of the objects (18) and the non- 
optical subsystem (12) is adapted to produce the 
position and/or orientation signals of another one 
(22) of the objects. 

4. A system according to claim 1 , characterized in that 
the coupling arrangement comprises: 

a non-optical subsystem light source (24) hav- 
ing a fixed relationship to the non-optical sub- 
system coordinate system (X M , Y M , Z M ); 
the optical subsystem sensor (20) is adapted to 
detect energy from the non- optical subsystem 
light source (24) and produce position and/or 
orientation signals in response to such non- 
optical subsystem light source detected energy 
relative to the optical subsystem coordinate 
system (Xq, Yq, Zq); and 
a processor (26) is responsive to signals pro- 
duced by the optical and/or non-optical subsys- 
tems for determining the position and/or 
orientation of one or more of the objects rela- 
tive to the coordinate system of the selected 
one of the optical or non -optical subsystems. 

5. A system according to claim 4, characterized in that 
the optical light sources (43) are mounted to one 



(41 ) of the objects and the non-optical subsystem 
(12) is adapted to produce the position and/or ori- 
entation signals of such one (41 ) of the objects. 

5 6. A system according to claim 4, characterized in that 
the optical light sources (16a, 16b, 16c) are 
mounted to one (18) of the objects and the non- 
optical subsystem (12) is adapted to produce the 
position and/or orientation signals of another one 

10 (22) of the objects. 

7. A system according to claim 1 , characterized in that 
the coupling arrangement comprises: 

15 a non-optical subsystem sensor having a fixed 

relationship to the optical subsystem coordi- 
nate system, such non-optical subsystem sen- 
sor being adapted to produce position and/or 
orientation signals of the optical subsystem 

20 light source relative to the non-optical subsys- 

tem coordinate system; and in that a processor 
is responsive to signals produced by the optical 
subsystem sensor and/or the non-optical sys- 
tem sensor for determining the position and/or 

25 orientation of one or more of the objects rela- 

tive to the coordinate system of the selected 
one of the optical or non-optical subsystems. 

8. A system according to claim 7, characterized in that 
30 the optical light sources are mounted to one of the 

objects and the non-optical subsystem is adapted 
to produce the position and/or orientation signals of 
such one of the objects. 

as 9. A system according to claim 7, characterized in that 
the optical light sources are mounted to one of the 
objects and the non-optical subsystem is adapted 
to produce the position and/or orientation signals of 
another one of the objects. 

40 

10. A system for determining spatial position and/or ori- 
entation of one or more objects, comprising: 

(A) an optical subsystem with light sources 
45 mounted to one or more of the objects; and a 

sensor adapted to detect energy from the opti- 
cal subsystem light sources; 

(B) a non-optical subsystem; 

(C) a coupling arrangement for producing posi- 
50 tion and/or orientation signals indicative of the 

position and/or orientation of the selected one 
of the optical or non-optical subsystems; and 

(D) a processor responsive to signals produced 
by the coupling arrangement and the optical 

55 and/or non-optical subsystem for determining 

the position and/or orientation of one of more of 
the objects. 
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11. A system according to claim 1 or 2 or 3, character- 
ized in that the said means for determining the posi- 
tion and/or orientation includes a processor (26). 
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101- 



Calibrate the optical subsystem 
sensor and fix its local frame of 
reference. Store the resultant 
operational parameters. 



102- 



103- 



Define the magnetic field 
generator as a rigid body in 
the optical subsystem local 
frame of reference and store 
the definition. 



Calibrate the magnetic 
subsystem and fix its local 
frame of reference. Store the 
resultant operational 
parameters. 



104- 



' Determine a suitable global frame of 
/reference for the entire hybrid system, 
as well as the transformations relating 
the subsystem local frames of 
reference to the global frame of 
reference, and store these. 



FIG. 3 
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207-J Position the magnetic field generator 
I at some convenient location within 
/ the optical subsystem operational 
/ volume, ensuring that a suitable 
/ number of reference markers are in 
/ the optical sensor's view j 



202- 



Perform the following 
sequence for each body 
being tracked. 



203- 



Measure the magnetic 
field generator's position 
relative to the optical 
sensor position. 



204 



Was 

Yes either the magnetic' 

" " field generator or the optical 

sensor moved 
? 



No 



Update the transforms 
between the subsystem local 
frames of reference and the 
global reference frame. 



205 



Z7 



No, 



® 



206z 

Are both 
'optical and magnetic data* 
available 

9 
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Determine the object 
position and pose from 
the optical sensors. 




Determine the object 
position and pose from 
the magnetic sensors. 


208 y 




210 y 




Apply the optical 
subsystem to global 
frame of reference 

transformation. 




Apply the magnetic 
subsystem to global 
frame of reference 
transformation. 



209- 



211 



FIG. 4B 
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212- 



© 

1 

Subsystem exchange data 
to initialize position 
determine algorithm or 
further refine the results 



213- 



I 



Determine the object position 
and pose from the optical 
sensors and estimate their 

uncertainty from the optical 
subsystem error map. 



214- 



I 



Transform the object position 
and pose to the global frame 
of reference using the optical 
subsystem frame of reference 
transformation. 



215- 



I 



Determine the object position 
and pose from the magnetic 
sensors and estimate their 
uncertainty from the magnetic 
subsystem error map. 



216- 



I 



217- 



Transform the object position 
and pose to the global frame 

of reference using the 
magnetic subsystem frame of 
reference transformation. 

I 



Combine the individual 
subsystem measurements 

according to their 
corresponding estimated 
uncertainties 

FIG. 4C 
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